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(54) RETRIEVAL DEVICE AND STORAGE DEVICE, AND RETRIEVING METHOD AND STORING 
METHOD FOR MUSIC INFORMATION, AND STORAGE MEDIUM WHERE PROGRAMS 
THEREOF ARE RECORDED 

(57)Abstract: 

PROBLEM TO BE SOLVED: To make the retrieval speed 
faster than that for an entire search by using indexes of 
feature quantities extracted from music information and 
to comply with a variety of retrieval requests by 
performing similar retrieval based upon feature 
quantities. 

SOLUTION: In the conversion and storage phase of a 
retrieval object, a conversion part 3 converts source 
data of music to music information, divides the music 
information into pieces of partial data, and extracts 
feature quantities from the pieces of partial data. A 
storage part 4 stores the extracted feature quantities of 
the retrieval object in a data base with indexes. In the 
retrieval phase of music information, the conversion part 
to music information 3 similarly extracts feature 
quantities from the source data of music as retrieval 
conditions and a music information retrieval part 5 
retrieves similar source data from the data base in the 
storage part 4 by making use of indexes. The result is 
outputted to a display and sound source device 6. 

♦ NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 
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CLAIMS 



[Claim(s)] 

[Claim 1]A conversion method to music information which changes into music information in 
which a computer can analyze source data of a musical piece in a retrieval device with which a 
musical piece is searched, divides the music information into two or more piece data, and 
extracts characteristic quantity from piece data, An accumulation means which accumulates as 
a database characteristic quantity extracted by a conversion method to the above-mentioned 
music information about source data of a musical piece used as a retrieval object, A music 
information search means to search source data which were similar from a database of the 
above-mentioned accumulation means using characteristic quantity extracted by a conversion 
method to the above-mentioned music information about source data of a musical piece used as 
a search condition. A music information retrieval device provided with a search-results output 
means which outputs search results. 

[Claim 2]In the music information retrieval device according to claim 1, the above-mentioned 
music information search means, A music information retrieval device searching source data 
which measure similarity and are similar to input data by it by carrying out distance calculation of 
characteristic quantity extracted from music information of input data specified as a search 
condition, and accumulated characteristic quantity. 

[Claim 3]In the music information retrieval device according to claim 1 or 2, a conversion method 
to the above-mentioned music information. When characteristic quantity extracted from music 
information is stored in a database, from those characteristic quantity, create an index and the 
above-mentioned music information search means, A music information retrieval device 
searching source data which were similar from the above-mentioned database based on 
characteristic quantity extracted from music information of source data using the above- 
mentioned index. 

[Claim 4]A music information retrieval device characterized by the above-mentioned indexes 
being one thru/or a multi-dimension index based on two or more characteristic quantity in the 
music information retrieval device according to claim 3. 

[Claim 5]In a music information storage device used for a retrieval device with which a musical 
piece is searched, change source data of a musical piece used as a retrieval object into music 
information which a computer can analyze, and the music information is divided into two or more 
piece data. While extracting characteristic quantity from piece data and storing in a database, A 
conversion method to music information which creates an index from those characteristic 
quantity, A music information storage device provided with an accumulation means which 
accumulates characteristic quantity extracted by a conversion method to the above-mentioned 
music information about source data of a musical piece used as a retrieval object as a database 
with an index. 

[Claim 6]An accumulation means which accumulates characteristic quantity which divided the 
music information into piece data, and was extracted from music information which changed 
source data of a musical piece used as a retrieval object in a retrieval device with which a 
musical piece is searched as a database with an index, A conversion method to music 
information which changes source data of a musical piece used as a search condition into music 
information which a computer can analyze, divides the music information into two or more piece 
data, and extracts characteristic quantity from piece data, A music information retrieval device 
provided with a music information search means to search source data which were similar from 
the above-mentioned database using the above-mentioned index based on characteristic 
quantity extracted from music information of the above-mentioned source data, and a search- 
results output means which outputs search results. 

[Claim 7]In a method of accumulating music information used for a retrieval device with which a 
musical piece is searched, While storing a process in which change source data of a musical 
piece used as a retrieval object into music information which a computer can analyze, divide the 



music information into two or more piece data, and characteristic quantity is extracted from 
piece data, and extracted characteristic quantity in a database, A music information 
accumulating method having a process in which an index is created from those characteristic 
quantity. 

[Claim 8]The music information is divided into piece data from music information which changed 
source data of a musical piece used as a retrieval object. Extracted characteristic quantity as a 
database with an index. Are a search method which searches a musical piece using an 
accumulation means to accumulate, change source data of a musical piece used as a search 
condition into music information which a computer can analyze, and the music information is 
divided into two or more piece data. A music information search method having a process in 
which source data which were similar from the above-mentioned database using the above- 
mentioned index based on a process in which characteristic quantity is extracted from piece 
data, and characteristic quantity extracted from music information of the above-mentioned 
source data are searched, and a process in which search results are outputted. 
[Claim 9]Are the recording medium which recorded a program for accumulating music information 
used for a retrieval device with which a musical piece is searched, change source data of a 
musical piece used as a retrieval object into music information which a computer can analyze, 
and the music information is divided into two or more piece data. A recording medium which 
recorded a music information stored program recording a program which makes a computer 
perform processing which extracts characteristic quantity from piece data, and processing which 
creates an index from those characteristic quantity while storing extracted characteristic 
quantity in a database. 

[Claim 10]The music information is divided into piece data from music information which changed 
source data of a musical piece used as a retrieval object. Extracted characteristic quantity as a 
database with an index. Are the recording medium which recorded a program for searching a 
musical piece using an accumulation means to accumulate, change source data of a musical 
piece used as a search condition into music information which a computer can analyze, and the 
music information is divided into two or more piece data. Processing which searches source data 
which were similar from the above-mentioned database using the above-mentioned index based 
on processing which extracts characteristic quantity from piece data, and characteristic quantity 
extracted from music information of the above-mentioned source data, A recording medium 
which recorded a music information retrieval program recording a program which makes a 
computer perform processing which outputs search results. 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention relates to the device which accumulates the music 
information used for the device and music information search which perform music information 



search with the source data of the sound source from humming, a keyboard, etc. 
[0002] 

[Description of the Prior Art]The note of the composition data which serves as an inputted 
search condition as the conventional method of performing music information search is taken, 
and there are some which perform collation (total search) with the composition data used as a 
retrieval object In this method, input data by FFT (Fast Fourier Transform: Fast Fourier 
Transform) or autocorrelation. From the start of the composition data used as the retrieval 
object which performs copying-down-a-tune processing changed into a score, and is stored in 
the database etc. by making a pitch and sound length into characteristic quantity to the end, It 
compares by one DP (Dynamic Programming) matching of a method which carries out pattern 
matching, and it is compared with the composition data used as all the retrieval objects, and let a 
correct data number (what has a near distance of a course) be search results. 
[Reference] 

- Tomoya Sonoda, Masataka Goto, and Yoichi Muraoka: "music retrieval system by singing 
voice", the 55th time of Information Processing Society of Japan national conference, 1 J-B- 
1997. 

- Tetsuya Kageyama, Takashima ****: "melody search which makes humming song key", 
Institute of Electronics, Information and Communication Engineers paper magazine, and Vol.J77- 
D-II No.8 pp.1 543-1 551, 1994. 

[0003]A score is made into numerals (character) and there are some which perform music 
information search by the method of transposing to the character string search which regards it 
as a character string and is known conventionally, and searching as shown, for example in JP,07- 
121556,A (music information retrieval device). 

[0004] Drawing 10 is a figure for explaining DP matching. DP matching is the technique of using 
dynamic programming for making the pattern of different length compare. A=a1, a2, — , ai, — , the 
course (polygonal line) that connects the lattice point amB=b1, b2, — , on the two-dimensional 
flat surface which makes both rectangular coordinates for the correspondence relation of each 
character of A and B when it is referred to as bn, bj, — , express the character string made to 
compare, respectively. A course which is in agreement with the both sides of the character 
strings A and B according to this course is determined. As a result, the long thing of similarity of 
the maximum coincident part string length will be near. In the example of drawing 10 . the course 
which connects the lattice point of a, b, c, d, and a shown with the round head will be chosen in 
collation of the pitch sequences A and B. 
[0005] 

[Problem(s) to be Solved by the Invention]In the conventional method mentioned above, about 
the score data of input data, and the score data stored in the database, in order to perform 
pattern matching from the start of data to the end, there were the following problems. 

(1) Retrieval speed is slow because of the search (exhaustive search) which is not using the 
index in the conventional search. 

(2) In the conventional search, since it is not searching using the features (the height of vocal, 
the thing corresponding to modulation, etc.) of music information, various retrieval required, such 
as "I would like to choose the high music of vocal" and for example, "liking (to choose music 
which was modulated)", cannot be met. 

[0006]An object of this invention is to enable it to meet various retrieval required by 
accelerating retrieval speed from exhaustive search, and carrying out similar retrieval with two or 
more characteristic quantity by solving the above-mentioned problem and using the index of the 
characteristic quantity extracted from music information. 
[0007] 

[Means for Solving the Problem]In order that this invention may solve an aforementioned 
problem, a means to change into music information source data explained below, a means to 
search from a database, and a means to output search results are used. 

[0008]A means to change into the above-mentioned music information changes source data of a 
musical piece into music information which can analyze a computer, and divides music 
information of the whole into two or more piece data. Next, it changes into music information for 



every frequency component using numeric conversion processing so that it may become possible 
to extract the musical feature from piece data. And characteristic quantity is extracted from 
music information for every frequency component, and it accumulates in a database. An index 
(multi-dimension index) is created from accumulated characteristic quantity. Here, since 
characteristic quantity serves as a multi dimensional vector, although vector space of many 
dimensions is searched, for example, indexes, such as convenient R-tree, are created. 
[0009]Dividing music information into piece data and a point of extracting two or more musical 
characteristic quantity which is called "height of vocal", for example from the music information 
differ from conventional technology. 

[001 0]A means to search from the above-mentioned database searches characteristic quantity 
extracted from music information equivalent to an input key as compared with characteristic 
quantity accumulated in a database. At this time, search performs similarity calculation which 
makes [ of similarity ] high what has a near distance using a distance function (similar retrieval). 
It searches using the above-mentioned index. 

[001 1]Performing search by similarity and a point of using an index of characteristic quantity 
differ from conventional technology. 

[0012]The above-mentioned result display means uses a display and a loudspeaker for a 
retrieving person by making a searched musical piece into search results, and enables viewing 
and listening of them. 

[0013]The operation of this invention is as follows. Source data (CD, a score, the live, etc.) 
based on various media can be systematically searched now with changing into music information 
which can extract the musical feature from a musical piece by a means to change into the 
above-mentioned music information. It becomes possible to extract two or more amounts of 
musical features. By having combined many amounts of musical features, it becomes possible to 
meet various retrieval required. It does not need to be accompanied by whole search by dividing 
the whole music information into piece data, and creating an index. 

[0014]Searching at high speed is possible by using an index (multi-dimension index) based on 
characteristic quantity extracted from music information by a means to search from the above- 
mentioned database. Music which resembled input data by similar retrieval which uses a distance 
function can also be searched, and it becomes possible to correspond also to various inquiries. 
[0015]The above-mentioned result display means enables it to tell a retrieving person about 
search results visually or auditorily. 

[001 6]A program for a computer to realize each above processing means is storable in suitable 
recording media, such as a portable medium memory which a computer can read, semiconductor 
memory, and a hard disk. 
[0017] 

[Embodiment of the Invention] Drawing 1 is a block diagram showing the example of composition 
of this invention. The musical piece input device 1 is a device which inputs the source data of 
musical pieces, such as playback equipment, such as an input device of music data, such as a 
microphone which inputs humming, singing voice, etc., and a score, a tape, and a compact disk 
(CD), or a keyboard for music. The search condition input device 2 may also be the same, and 
the musical piece input device 1 and the search condition input device 2 may be the same 
devices. 

[0018]The converter 3 to music information is a means to change source data into music 
information, to extract two or more characteristic quantity from the music information, and to 
store in the database of the accumulating part 4. It differs from conventional technology in that 
the converter 3 to this music information performs division into piece data, extraction of 
characteristic quantity, and creation of an index. 

[0019]The accumulating part 4 is a database with the information on the characteristic quantity 
obtained from the music information used as a retrieval object, and the index (multi-dimension 
index) to them. 

[0020]The music information retrieval part 5 is a means to search the source data which were 
similar from the database of the accumulating part 4 using the characteristic quantity extracted 
from the music information of the source data used as a search condition by the converter 3 to 



music information. The point of performing similar retrieval which measures similarity by distance 
calculation, such as not DP matching but Euclidean distance and a Manhattan distance, and the 
point of using an index differ from the conventional music information search. 
[0021]A display and the sound source device 6 are devices which output the result searched by 
the music information retrieval part 5, and are a sound source device, loudspeakers, etc., such as 
a display which displays a track name or a score, or MIDI. 

[0022]Here. the source data of a musical piece are what was inputted from music data, humming, 
singing voice, the keyboard for music, etc., scores, these one, or two or more things. The music 
information which can analyze a computer means what expresses the source data themselves 
with the data format which a computer tends to process like the digital data which sampled 
music data. In particular, in this embodiment, the inputted source data are changed into the 
music information of the form systematically defined inside by the converter 3 to music 
information, even if image data, such as an analog signal, MIDI data, and a score, etc. are what 
kind of things. 

[0023] Hereafter, the example of each part of the above is explained in more detail. 
[0024][The conversion phase to music information] Drawing 2 is a flow chart of the conversion 
phase to music information. First, source data are inputted from the musical piece input device 1 
(Step S10). Next, the source data inputted from the musical piece input device 1 are sampled 
and evaluated with the low rate near 8 kHz of a monophonic recording by the converter 3 to 
music information (Step S11). Next, the digital data which is the music information of the 
evaluated whole is divided into two or more piece data (Step S12). 

[0025]The method of division is divided into the piece data for tens of seconds, shifting digital 
data several seconds respectively, as a lap is made. The example of the division into piece data 
is shown in drawing 3 . As shown in drawing 3 . let division of music information be the slide 
window (Sliding window). 

[0026]Next, FFT with a window is given to the divided piece data. A window uses the Hanning 
window, a humming window, etc. The knot portion by the periodicity of FFT is made not to 
become discontinuous by using this window. Shifting the digital data of a limited individual (for 
example, every 512 pieces) little by little, FFT is taken until data is lost, it is given to each, and 
creates the new music information for extracting characteristic quantity. Drawing 4 (A) receives 
the signal of the input data about piece data, drawing 4 (B) receives the input data of drawing 4 
(A), and the example of the music information acquired by giving FFT is shown. 
[0027] Next, data is gathered at intervals of logarithm and the characteristic quantity used as the 
feature is extracted from the compass (50 Hz - near 20000 Hz) of only the range which hears 
human being as a vector. The extracted feature amount vector is stored in the accumulating part 
4, and an index (index) is created so that it may mention later (Step SI 3). 
[0028]Here as an example of a feature amount vector. That in which the frequency used as the 
peak of the sound pressure of **, the sound pressure ingredient of (1) of the whole, and (2) 
frequency components, the frequency used as the peak of the sound pressure of the frequency 
component which normalized (3) frequency at a certain constant interval, and frequency with 
high sound pressure obtained from (4) above (3) took difference serially, ******. 
[0029]When making the sound pressure ingredient of (1) of the whole into a feature amount 
vector, the number of the data gathered for every frequency is histogram-ized for every unit of 
the, and it is considered as a vector-dimensional [ several ]. The example which makes the 
whole sound pressure ingredient the feature amount vector of n dimension is shown in drawing 5 . 
In drawing 5 , a one-dimensional eye is the total of the sound pressure in the range of 0 to 1 00 
Hz, and this is equivalent to the area of the portion which gave HATCHIGGU into the figure. 
[0030]By music information, in the case of (2), the number of the data of the maximum is 
histogram-ized for every summarized unit, and let it be a vector^dimensional [ several ]. The 
example of the position of the peak of the sound pressure made into characteristic quantity here 
is shown in drawing 6 . Drawing 6 (A) shows the peak acquired from the music information of the 
search key, and the peak in the music information of the musical piece accumulated in the 
database and drawing 6 (B) can judge it to be similar music information, when these histograms 
are in agreement with many piece data. A histogram is made by counting the number of the 



greatest peak to each section which gives FFT mentioned above in each piece data and which 
made for every unit, for example, divided frequency at intervals of logarithm. 
[0031]In the case of (3), music information is folded up on frequency at a constant interval 
(collecting), and the number of the greatest data in the section is histogram-ized for every 
summarized unit, and let it be a vector-dimensional [ several ]. 

[0032] For example, when folding up among 100 to 200 Hz, the music information a is :a=a[ when 
larger than 200 (Hz) ] /2 (it repeats until it becomes smaller than 200). 

The music information a is :a=a[ when smaller than 100 (Hz) ] x2 (it repeats until it becomes 100 
or more). 
It carries out. 

[0033]In the case of (4), difference with the next data of the data folded up on the frequency of 
the above (3) at the constant interval is summarized for every difference, the number of the data 
is histogram-ized, and it is considered as a vector-dimensional [ several ]. The example is shown 
in drawing 7 . At 50 Hz. as shown in drawing 7 , if the frequency of the data to which its attention 
was paid changes to 55 Hz (movement), it will add the difference (5) to the section when the 
vector of n dimension is applied as 1 (piece), and will unite it with it in following time (for 
example, after 2 seconds) (it counts). This section is the section which divided the difference 
value at intervals of logarithm, for example, in the section from 1 00 Hz to 200 Hz. The maximum 
of a difference is 100, to each section which divided this at intervals of the logarithm 0 to 2 and 
2 to 4 and 4 to 8, and — , creates a histogram and makes this the feature amount vector of n 
dimension. 

[0034]As mentioned above, each characteristic quantity serves as a multi dimensional vector. 
Euclidean distance and a Manhattan distance are used for the similarity calculation by this multi 
dimensional vector. The changed characteristic quantity is stored in the database of the 
accumulating part 4. 

[0035]A separate index is created for every characteristic quantity based on the characteristic 
quantity extracted from real-intention easy information, respectively. Even if it uses the index of 
what kind of form, it is feasible, but this invention is suitable to search the vector space of many 
dimensions if indexes, such as R-tree, are used, for example. In this method, grouping of the 
vector of the characteristic quantity in multi^dimension space is carried out by what are 
mutually similar, this is repeated, a tree structure is completed, and an index is created. Since 
this can limit access only to the data considered for evaluation to be required by following a 
layered structure, even if a data number increases, it can maintain search speed of response. 
There are the following as a reference about this R-tree. 

[Reference] A.Guttman, "R-tree : a dynamic index structures for spatialsearching". Proceedings 
of the ACM SIGMOD International Conference on the Management of Data, pp.47-57, Boston, 
1984. [The search phase of the music information from a database] Drawing 8 is a flow chart of 
the search phase of the music information from a database. First, the key data (source data) 
which serves as a search condition from the search condition input device 2 is inputted (Step 
S20). This input data is changed into music information by the converter 3 to music information, 
and characteristic quantity is extracted (Step S21). Next, by the similar retrieval using an index, 
the characteristic quantity obtained from key data and the characteristic quantity accumulated 
in the database of the accumulating part 4 are measured, and source data are searched with the 
music information retrieval part 5 (Step S22). Search results are displayed on a display and the 
sound source device 6 as a result, or are reproduced using a sound source device. 
[0036]In the similar retrieval in the music information retrieval part 5, the distance calculation of 
the characteristic quantity of the input data to search and the characteristic quantity of the 
piece data accumulated in the database performs similar retrieval. Here, by attaching dignity and 
carrying out linear combination of the intermediate result which was able to obtain the 
characteristic quantity of music information from the index according to each characteristic 
quantity for every characteristic quantity, comprehensive similarity is calculated and similar 
retrieval is performed based on it. At the time of search, it searches using the index created 
based on characteristic quantity. 

[0037] Drawing 9 is a figure for explaining the example of the similar calculation method in this 



embodiment. As shown in drawing 9 (A), it compares with the characteristic quantity in the 
database which extracts various kinds of characteristic quantity from the music information 
acquired from an input key, and is extracted from the source data of the retrieval object The 
characteristic quantity of the source data in a database is distributed all over multi-dimension 
space, as shown in drawing 9 (C). and as the index which clustered and tree-structure-ized 
these shows drawing 9 (B). it is created So, when searching the data (characteristic quantity) of 
a database, it narrows down using this index, the distance of the data (characteristic quantity) 
which is there is compared, and it returns as a result sequentially from a near thing. 
[0038]In the similar retrieval using two or more above characteristic quantity, search of the 
musical piece for which a user asks is easily realizable by attaching the dignity according to the 
user's retrieval object for every characteristic quantity, and performing similar calculation. For 
example, it can respond making heavy dignity of the characteristic quantity of (2) explained as an 
example of a feature amount vector to the retrieval required of "liking to choose the high music 
of vocal", i.e., by enlarging the weight value applied to the value of characteristic quantity. It can 
respond by making heavy dignity of the characteristic quantity of (3) explained as an example of 
a feature amount vector, and (4) to the retrieval required of music which was modulated. 
[0039] 

[Effect of the Invention]As mentioned above, according to this invention, even if the retrieving 
person does not have special musical knowledge or skill, the musical piece from humming, a 
sound source, a keyboard, etc. enables it to search music at high speed from a database. Similar 
retrieval enables it to correspond (for example, music etc. which were modulated) at various 
inquiries. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is a block diagram showing the example of composition of this invention. 

[Drawing 2] It is a flow chart of the conversion phase to music information. 

[Drawing 3] It is a figure showing the example of the division into the piece data of music 

information. 

[Drawing 4] It is a figure showing the example which gave FFT in the divided piece data. 
[Drawing 5] It is a figure explaining the example which makes the whole sound pressure ingredient 
the feature amount vector of n dimension. 

[Drawing 6] It is a figure showing the example of the position of the peak of the sound pressure 
made into characteristic quantity. 

[Drawing 7] It is a figure explaining generation of n dimension feature amount vector which paid 
its attention to transition of data. 

[Drawing 8] It is a flow chart of the search phase of the music information from a database. 
[Drawing 9] It is a figure for explaining the example of the similar calculation method in this 



embodiment. 



fDrawing 10] It is a figure for explaining the conventional DP matching. 
[Description of Notations] 

1 Musical piece input device 

2 Search condition input device 

3 The converter to music information 

4 Accumulating part 

5 Music information retrieval part 

6 A display and a sound source device 
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